Differential Contrastive Divergence
نویسنده
چکیده
We formulate a differential version of contrastive divergence for continuous configuration spaces by considering a limit of MCMC processes in which the proposal distribution becomes infinitesimal. This leads to a deterministic differential contrastive divergence update — one in which no stochastic sampling is required. We prove convergence of differential contrastive divergence in general and prove convergence to the optimal parameter value under natural but restrictive assumptions. Contrastive divergence [2, 1] is a method for density estimation based on setting the parameters of a Gibbs distribution. More formally, let x be a variable ranging over a continuous configuration space and consider a Gibbs distribution on x where the energy is parameterized by a parameter vector β. P (x;β) = 1 Z(β) e
منابع مشابه
Stochastic Gradient Estimate Variance in Contrastive Divergence and Persistent Contrastive Divergence
Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) are popular methods for training Restricted Boltzmann Machines. However, both methods use an approximate method for sampling from the model distribution. As a side effect, these approximations yield significantly different biases and variances for stochastic gradient estimates of individual data points. It is well known tha...
متن کاملDissimilarity Based Contrastive Divergence for Anomaly Detection
This paper describes training of a Restricted Boltzmann Machine(RBM) using dissimilarity-based contrastive divergence to obtain an anomaly detector. We go over the merits of the method over other approaches and describe the method’s usefulness to obtain a generative model.
متن کاملAverage Contrastive Divergence for Training Restricted Boltzmann Machines
This paper studies contrastive divergence (CD) learning algorithm and proposes a new algorithm for training restricted Boltzmann machines (RBMs). We derive that CD is a biased estimator of the log-likelihood gradient method and make an analysis of the bias. Meanwhile, we propose a new learning algorithm called average contrastive divergence (ACD) for training RBMs. It is an improved CD algorith...
متن کاملLearning Multi-grid Generative ConvNets by Minimal Contrastive Divergence
This paper proposes a minimal contrastive divergence method for learning energy-based generative ConvNet models of images at multiple grids (or scales) simultaneously. For each grid, we learn an energy-based probabilistic model where the energy function is defined by a bottom-up convolutional neural network (ConvNet or CNN). Learning such a model requires generating synthesized examples from th...
متن کاملLearning with Blocks: Composite Likelihood and Contrastive Divergence
Composite likelihood methods provide a wide spectrum of computationally efficient techniques for statistical tasks such as parameter estimation and model selection. In this paper, we present a formal connection between the optimization of composite likelihoods and the well-known contrastive divergence algorithm. In particular, we show that composite likelihoods can be stochastically optimized b...
متن کامل